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(57) Abstract: In a video camera surveillance 
system, a video processor determines dense 
motion vector fields between adjacent frames 
of the video. From the dense motion vector 
fields moving objects are detected and 
objects undergoing unexpected motion are 
highlighted in the display fo the video. To 
distinguish expected motion from unexpected 
motion, dense motion vector fields are stored 
representing expected motion and the vectors 
representing the moving object are compared 
with the stored vectors to determine whether the 
object motion is expected or unexpected. In an 
alternative embodiment, the video surveillance 
system comprises a panning camera and the 
frames of the video are arranged in a mosaic. 
Object motion in the video is detected by 
means of dense motion vector fields and the 
predicted position of objects in the mosaic is 
detected based on the detected object motion. 
The position of moving objects in the current 
frame being detected by the panning camera 
is compared with the predicted position of 
the objets in the mosaic and if the positions 
are substantially different, the corresponding 
object is tagged and highlighted as undergoing 
unexpected motion. A system is also disclosed 
for using the dense motion vector fields to 
control the motion of the panning camera to 
follow a moving object. 
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Surveillance Video Camera Enhancement System 



Background of the Invention 
This invention relates to surveillance video camera systems and more 
particularly to surveillance video camera systems enhanced by detecting object 

5 motion in the video to reduce overload on the operator's attention. 

In typical video camera surveillance systems of the prior art, multiple 
cameras are focused on multiple scenes and the resulting videos are transmitted to 
a monitoring area where the videos can be observed by the operator. The 
resulting multiple video motion pictures are simultaneously displayed or are 

1 0 displayed in sequence and it is difficult for the operator to detect when a problem 
has occurred in the detected scenes because of the large number of scenes which 
have to monitored In some systems, a video camera is pamed to increase the 
area that is monitored by a given camera. While such a system provides 
surveillance over a wide area, only part of the wide area is actually viewed at a 

15 given time leaving an obvious gap in the security provided by the scanning 

camera. To combat this latter problem, one system of the prior art combines the 
frames generated by the scanning camera into a mosaic so that entire scanned 
scene is displayed to the operator as an expanded panoramic view. In this system, 
each new video frame is compared with the previous detected frames displayed in 

20 the panoramic view and any differences are outlined thus providing an indication 
to the operator that the position of an object in the panoramic scene has changed. 
This system, while an improvement, nevertheless leads to an overload on the 
operator's attention, smce all objects in this panoramic scene which undergo a 
change in position will be outlined and it is still difficult for the operator to 

25 recognize tiiat one or more of tiie changes may represent a problem which requires 
attention. In addition, the fact that an object has undergone a change in position in 
many instances will not be brought to the operator's attention until the camera has 
completed a scannmg cycle and tiien only if the object location of an which is 
undergoing a change in position appears in two different frames in a scanning 

30 cycle. Accordingly, there is a need for a video camera system which unmediately 
brings to the operator's attention any significant or unexpected motion, which 
might represent a security problem requiring the operator's immediate attention. 
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Summary of the Invention 
In accordance with the present invention a video camera surveillance 
system is provided with a video processor which has the capability of immediately 
detecting any object motion in a detected scene and more particularly detecting 
5 the occurrence of unexpected motion in a detected scene. In accordance with one 
embodiment, a plurality of surveillance cameras are provided which feed the 
videos to a video processing system wherein the videos are analyzed to determine 
dense motion vector jfields representing motion between the frames of each video. 
From the dense motion vector fields, the motion of individual objects in the 

1 0 detected scenes can be determined and highlighted so that they are brought to the 
operator's attention. In accordance with the invention, the video processor stores 
dense motion vector fields representing expected motion in a scene and the dense 
motion vector field detected from the monitored scene is compared with the stored 
dense motion vector field representing expected motion to determine whether or 

15 not any unexpected motion has occurred. If an object is undCTgoing unexpected 
motion, this object will be highlighted in a display of the monitored scene. 

In accordance with another embodiment of the invention, the smveillance 
system comprises a panning camera which pans a wide scene. The frames 
produced by the video camera are combined into a mosaic representing a 

20 panoramic view of tihe scene scanned by the camera. By means of dense motion 
vector fields, object motion in the scene being monitored is detected and, based on 
the detected motion of the objects, the fixture movement of the objects is 
predicted- In those portions of the scaimed scene not currently being detected by 
the video camera, the position of the objects undergoing motion is updated in 

25 accordance with the predicted motion. Thus the moving objects in the panoramic 
scene will all be shown imdergoing motion and changing position in accordance 
with the predicted motion. As each new frame of the scanned scene is detected by 
the video camera, the mosaic is updated with the new frame. If a given object 
undergoing motion in the current detected scene is substantially displaced from 

30 the predicted position when the current detected frame containing such object 
updates the panoramic scene, such object is tagged as imdergoing unexpected 
motion and the object is highlighted. 

2 



PCT/USOl/42912 

WO 02/37856 

^Kp^ of Jsoen. wm.e ^ — ^ 



scene. 



InmeiJiate attention and action. 

Brief Description of the Drai»rings 
, is a bio* diagtan. of a s^dHanc. ^deo c^nera system in 
^riancemth one embodimait of the invention. .„„,ta*e 
15 ji^2isaflowch»tinust,a«ngmevideoproc«sm8camedo»,bytt.e 

videoprocessingsystemintheembodin^tof figure 1. 

Kg„.3innstra.e,adisplaycrea.edbyfl.systemmfig«elwhere^aa 

^^g„^tn>aybebigbii*.edbysbo»^gatelescopiceni»gedv,ewotan 

area around a moving object .j..,i,v.he 
20 Fig<.rc4iUus.r«es.notodisplaywbichmaybeprov.dedby*e 

surveiUance system shown in figure 1. . . , 

Hj^Sisabloc^aiagtamofanoto^bodimentoftbepresen. 

enroloying a scanning video camera. 

6 i» - illustration of a tnos^c display create by *e syst^ of 

"^«^7isanowchartiUustta.ing.beprocesscatriedontby*evideo 

processor by the system in figure 5. 

Description of Preferred Embodiments 

InthesystemoftheinventionasshowninFigurelapIuralityofvi^^^^^^ 

30 camerasnarecacharr^gedtodetectavideoimageofanareatobemon.or^^^^ 

30 cameras .,l.r,f.e svstem Bach camera will send a sequence of 

by the video camera surveillance system. c<^^ 

3 



wo 02/37856 PCTAJSO 1/4291 2 

video frames showing the corresponding monitored area to a video data 
processing system 13. The video data processing system typically will comprise a 
video processor for each video camera but a high speed video processor could be 
employed to process the sequence of video frames received from each of the 
5 cameras simultaneously. The video data processing system detects object motion 
represented in the video received from each camera, highlights selected moving 
objects in the video, and transmits the resulting video to a video display system 15 
• in which the video frx>m the four cameras are displayed. The video display system 
may be a separate monitors to display the videos simultaneously, or the videos 

1 0 may be all simultaneously displayed on the screen of the single monitor. 

Alternatively, the videos from the separate cameras may be displayed in sequence 
on one or more video monitors. 

In preferred embodiments, the video processor will detect unexpected 
motion of objects in the videos and will highlight the objects undergoing this 

1 5 unexpected motion. Obj ects undergoing motion which is e3q)ected, may be 

highligjited in a diJBFerent way flian the way the imexpected motion is highlighted. 

A flow chart of the process carried out by the video processing system 13 
on a video received from one of the cameras in shown in Figure 2, As shown in 
this Figure, the video from one of the cameras is fibrst processed to detect dense 

20 motion vector fields representing the motion of image elements in the received 
video. Image elements are pixel sized components of objects depicted in the 
video and a dense motioii vector field comprises a vector for each image element 
indicating the motion of the corresponding image element A dense motion vector 
field will be provided between each pair of adjacent frames in the video 

25 represmting the motion in the video from frame to frame. The dense motion 
vector fields are preferably generated by the process disclosed in co-pending 
application Serial No. 09/593,521 entitled "System for the Estimation of Optical 
Flow'*, filed June 14, 2000 and invented by Sigfiiend Wonneberger, Max Griessl 
and Markus Wittkop. This apphcation is hereby incorporated by reference. 

30 From the dense motion vector fields, the moving objects in the video are 

identified and are selectively highlighted. In a simplified version of the invention, 
all moving objects could be highlighted simply by changing a characteristic of all 
of the pixels in each video frame corresponding to a niotion vector having a 

4 
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could be left unchanged and unhighlighted. When the detected object motion 
does not correspond to expected motion as represented by the stored dense motion 
vector fields, the object undergoing the unexpected motion is highlighted in a 
different way such as being tinted red or surrounded by a halo, or alternatively as 
5 being subjected to a telescopic effect. In producing the telescopic effect, the video 
processor defines a high resolution viewing bubble around the object undergoing 
the unexpected motion or around the area of the xmexpected motion and magnifies 
it as shown in Figure 3. The operator may be given the ability to electronically 
steCT the magnified viewing bubble around the scene to more clearly view items of 

10 interest. In a preferred embodiment, the unexpected motion is automatically 

highlighted by changing the pixel characteristic such as color or by adding a halo 
and then tihe operator can optionally define the high-resolution viewing bubble 
aroimd the highligjited object after having his attention drawn to unexpected 
motion by the original highlighting. 

15 In addition, in the preferred embodiment, the video processor can exclude 

firom the highlighting process any trivial motion such as a motion of a small 
magnitude or a motion of a small object such as that of a small animal. 

In the system as described above, the object having unexpected motion 
may be highlighted by changing its color, by changing its saturation, by changing 

20 it contrast, or by placing a halo around the object. Alternatively, the object may 
be highlighted by defocusing the backgroimd which is not undergoing unexpected 
motion or by changing the background to a grey scale depiction. 

Another feature of the present invention is illustrated in Figure 4, In 
accordance with this feature of the invention, a moving object in the display is 

25 identified as described above by flagging the contiguous pixels representing the 
moving object. The velocity of the moving object is then detected from the dense 
motion vector field vectors representing the motion of the picture elements 
corresponding to the moving object. Information is then added to the display to 
indicate the speed and direction of the moving object as shown in Figure 4. The 

30 information may be in the form of an arrow indicating the direction of the motion 
and containing a legend in tiie arrow indicating the speed and feet per second and 
the heading of the object in degrees. In Figure 4, the cart being pushed by a 
customer is moving at 2.6 feet per second at a heading of 45"*. 
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by changing its color, by changing its brightness or saturation, by placing a halo 
around the object, or by magnifying the area around the object to provide a 
telescopic effect at the location of the object 

The camera is panned to scan the scene at a slow enough rate that so each 
5 location, is detected in several sequential frames during the scan of the camera. 
This feature mables the system of the invention, making use of dense motion 
vector fields to detect object motion, to detect any object motion during each scan 
of the seme. 

The system of figure 5 may also detect unexpected motion by storing 

1 0 dense motions vector fields representing expected motion in the maimer described 
above in connection with the embodiment of figure 1. Because the system of 
figure 5 detects object motion inmiediately, this form of une3q)ected motion may 
be inmiediately highlighted without waiting for the camera to again cycle through 
the same portion of the scene. 

15 As a result of viewing the entire seamed scene, including predicted object 

motion in the scene, the operator may wish to get an immediate update of a 
specific object in the scanned scene. Rather that wait for the panning camera to 
again reach the object, the operator can cause the camera to snap to that view by 
means of servomechanism 27 for a real time display of the object of interest and 

20 can cause die camera to zoom in on the object if desired. 

Figure 7 is a flow chart illustrating the operation of the video processor to 
make a mosaic of the received picture firames to display the entire scene scanned 
by the camera and to detect moving objects and to predict and display their 
predicted positions in the scaimed scene. As shown in Figure 5 the video is first 

25 processed in step 3 1 to detect the dense motion vector fields representing the 

motion of image elements between the currently detected firame and the adjacent 
firames in the video. Since the camera is being panned, the dense motion vector 
field will represent the apparent motion of the background due to the camera 
motion as well as motion of objects relative to the scene background. From the 

30 dense motion vector fields, the camera motion is detected and the motion of 

objects, separated fi'om the camera motion, is also detected in step 32. To detect 
the camera motion from the dense motion vector fields, the predominant motion 
represented by the vectors is detected. If most of the vectors are parallel and of 

8 
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As described above, objects undergoing unexpected naotion are 
highlighted. In addition, any objects undergoing expected motion or undergoing 
substantial expected motion may be highUghted in a different manner to 
distinguish them from objects undergoing imexpected motion as described above 
5 in connection with the embodiment of Figure 1 . 

As described above, the panning camera may be zoomed in and out. 
While the camera is being zoomed in and out, the action of the camera is 
considered camera motion and the video frames produced during the zooming or 
while the camera is. in a zoomed in or out state, can be added to the mosaic, hi 

1 0 this process the zooming camera motion is detected by the prevailing motion 
vectors extrading radially inwardly or outwardly. Once the camera motion has 
been detected, the size of the camera frames are adjusted to correspond to that of 
the mosaic frames and the cunrently detected frames are then located in the mosaic 
in the same manner as described above in connection with locating the camera 

1 5 frames produced by the camera panning motion. 

Tn accordance with anotha: feature of the inveation, the video processor, 
controls the operation of servomechanism 27 to cause the panning motion of the 
camera to follow a moving object and keep the moving object cmtered in the 
detected frame. To carry out this control, the video processor determines the 

20 predicted immediate fiiture locations of the moving objects. The predicted 

immediate future locations of the moving object are determined from the dense 
motion vector field vectors for the moving objects as explained above. By 
continuously moving the camera to the predicted immediate future locations of the 
moving object, the camera is made to follow the moving object keeping it 

25 centered in the currently detected frame. 

In the above described systems, the location of where the videos are 
displayed may be at a position a long distance from the position of the 
surveillance cameras. In such an instance, to permit the data to be transmitted 
over the long distance by telephone line or by the intemet, the transmitted data is 

30 compressed. In accordance with one embodiment, the video data is processed by 
a video processor at the location of the surveillance camera or cameras to identify 
and tag moving objects. Then after video has been transmitted to the display 
device representing the background being televised by the surveillance camera, 

10 
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WHAT IS CLAIMED: 

1. A surveillance method comprising detecting a scene to be 
surveilled with a video camera, processing the resulting video to detect moving 
objects in the detected scene, distinguishing objects undergoing unexpected 
S motion from objects undergoing expected motion and highlighting the objects 
imdergoing expected motion. 



2. A surveillance method as recited in claims 1 further comprising 
storing representations of motion expected in the detected scene and comparing 
the motion of moving objects in the detected scene with the stored representations 

10 of expected motion to distingiiish imexpected motion from expected motion. 

3. A method as recited in claim 1 further comprising: 
predicting the ftiture positions of the detected moving objects; 

comparing the actual positions of the moving objects with the predicted positions; 
and 

15 identifying an object as undergoing unexpected motion when the positions of such 
object is substantially different than the predicted position for such object. 

4- A method as recited in claim 1 wherein the objects undergoing 
unexpected motion are highlighted by magnifying the object and the area aroimd 
the object identified as undergoing unexpected motion. 

20 5. A surveillance method comprising detecting a scene to be 

surveilled with a video camera, detecting dense motion vector fields representing 
the motion of image elements from frame to firame in the video produced by said 
video camera, identifying moving objects depicted in said video by means of said 
dense motion vector fields and displaying said video with the identified moving 

25 objects highlighted in the display of said video. 



12 
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11. A surveillance system comprising a video camera arranged to 
detect a scene to be survielled, a video processor operable to detect moving 
objects in the video produced by said video camera and to distinguish objects 
undergoing unexpected motion from objects undergoing expected motion and to 

5 highlight the objects undergoing unexpected motion, and a display device 

connected to receive the processed video from the video processor to display the 
processed video with the objects undergoing xmexpected motion highlighted. 

12. A surveillance system as recited in claim 1 1 wherein said video 
processor is provided v/ith a storage capacity which stores representations of 

10 expected motion in a detected scene, said video processor comparing the motion 
of moving objects in a detected scene with the stored representations of expected 
motion to distinguish unexpected motion from expected motion. 

13. A surveillance system as recited in claim 1 1 wherein said video 
processor is operable to predict the future positions of the detected moving objects 

15 from the motion of the detected moving objects, to compare the actual positions of 
the moving objects with the predicted positions, and to identify an object as 
undergoing xmexpected motion when the actual position of such object is 
substantially different than the predicted position for such object. 

14. A surveillance system as recited in claim 1 1 wherein said video 

20 processor highlights an object undergoing unexpected motion by magnifjdng such 
object and the area around such object identified as undergoing imexpected 
motion. 

15. A surveillance system comprising a video camera arranged to 
detect a scene to be surveilled to produce a video of said scene, a video processor 

25 comiected to receive said video and operable to detect dense motion vector fields 
representing the motion of image elements from frame to frame in said video and 
to identify moving objects depicted on said video by means of said dense motion 
vector fields, and to highlight in said video a detected moving object by 
magnifying the display of the moving object and the area around moving object, 

14 



PCTAJSOl/42912 

WO 02/37856 

and a video display device coimected to receive the processed video from said 
video processor and to display the processed video with the moving object and the 
area around the moving object being magnified. 

16. A surveillance comprising a panning video camera arranged to 
5 detect a surveilled scene, a video processor coimected to receive the video 

produced by said video camera and operable to combine the frames of said video 
into a mosaic represeating a panoramic view of said scene, to detect the motion of 
moving objects in said scene, to determine the predicted position of objects in said 
scene determined from the detected motion of said objects and to update the 
position of said moving objects in said video in accordance with the predicted 
motion of said objects in said mosaic, and a video display device connected to 
receivethevideoprocessedby said video processor and to display saidmosaic as 
a panoramic view of the surveilled scene with the moving objects depicted in their 
predicted positions. 
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17. A surveillance system as recited in claim 16, wherein said video 
processor is further operable to compare the positions of detected moving objects 
in the frame currently detected by said video camera with the predicted positions 
of said moving objects, to flag a moving obj ect as having said unexpected motion 
whea the position of such moving obj ect in the cuireat frame is substantially 
different than the predicted position for such object, and to highlight in said 
mosaic the obj ect flagged as having unexpected motion whereby the display 
device displays said mosaic with the object undergoing unexpected motion 
hi^ighted. 

18. A video system comprising a panning video camera a video 
25 processor comiected to receive the video produced by said pamiing video camera 
and operable to generate dense motion vector fields representing the motion of 
image elements from frame to frame in the video produced by said video camera, 
to determine the motion of any depicted moving object in said video from said 
dense motion vector fields, and to predict the immediate fixture position of said 
30 moving object from the detected motion of said movmg object, a controUer for 
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controlling the motion of said video camera, said video processor controlling said 
controller to control the motion of said video camera in accordance with the 
predicted immediate future position of said moving object to maintain the moving 
object centered in the frame currently being detected by said video camera. 
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